Electronic device and method for controlling same

ABSTRACT

An electronic device is disclosed. The electronic device may comprise a first image sensor, a second image sensor, and a processor, wherein the processor may: acquire a first depth image and a confidence map by using the first image sensor; acquire an RGB image by using the second image sensor; acquire a second depth image on the basis of the confidence map and the RGB image; and acquire a third depth image by composing the first depth image and the second depth image on the basis of the pixel value of the confidence map.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a bypass continuation of International ApplicationNo. PCT/KR2021/008433 designating the United States, filed on Jul. 2,2021, in the Korean Intellectual Property Receiving Office and claimspriority from Korean Patent Application No. KR 10-2020-0094153, filed onJul. 29, 2020, the disclosures of which are incorporated by referenceherein in their entireties.

BACKGROUND 1. Field

The disclosure relates to an electronic device and a method forcontrolling the same, and more particularly, to an electronic device foracquiring a depth image and a method for controlling the same.

2. Description of Related Art

In recent years, with the development of electronic technology, researchon autonomous driving robots has been actively conducted. For smoothdriving of the robot, it is important to obtain accurate depthinformation about the robot's surroundings. In order to acquire depthinformation, time of flight (ToF) sensors that acquire a depth imagebased on flight time or phase information of light, or a stereo camerasthat acquire a depth image based on an image captured by two cameras maybe used.

However, the ToF sensors and the stereo cameras may have the followingdrawbacks. For example, while the ToF sensors have superior angularresolution for a long distance compared to the stereo camera, the ToFsensors have a limitation in that the accuracy of near-field informationis relatively low due to multiple reflections. On the other hand,although the stereo cameras may acquire short-distance information withrelatively high accuracy, two cameras need to be far apart from eachother for long-distance measurement, so the stereo cameras have thedisadvantage of being difficult to manufacture small in size.

Accordingly, there is a need for a technique for acquiring a depth imagewith high accuracy of near-field information while being easy tominiaturize.

SUMMARY

The disclosure provides an electronic device that is easy to miniaturizeand has improved accuracy of distance information for a short distance.

Objects of the disclosure are not limited to the above-mentionedobjects. That is, other objects that are not mentioned may be obviouslyunderstood by those skilled in the art from the following description.

According to an aspect of the disclosure, there is provided anelectronic device, including: a first image sensor; a second imagesensor; and a processor configured to: obtain a first depth image and aconfidence map corresponding to the first depth image based oninformation received from the first image sensor, obtain an RGB imagecorresponding to the first depth image based on information receivedfrom the second image sensor, obtain a second depth image based on theconfidence map and the RGB image, and obtain a third depth image basedon a composition of the first depth image and the second depth imagedetermined based on a pixel value of the confidence map.

The processor may be further configured to obtain a grayscale image fromthe RGB image, and the second depth image is obtained by performingstereo matching on the confidence map and the grayscale image.

The processor may be further configured to obtain the second depth imageby performing stereo matching on the confidence map and the grayscaleimage based on a shape of an object included in the confidence map andthe grayscale image.

The processor may be further configured to: determine a firstcomposition ratio value of the first depth image and a secondcomposition ratio value of the second depth image based on the pixelvalue of the confidence map, and obtain the third depth image bycombining the first depth image and the second depth image based on thefirst composition ratio value and the second composition ratio value.

The processor may be further configured to: determine the firstcomposition ratio value of the first depth image to be greater than thesecond composition ratio value of the second depth image for a firstregion in which a pixel value is greater than a reference value among aplurality of regions of the confidence map, and determine the firstcomposition ratio value to be smaller than the second composition ratiovalue for the region in which the pixel value is smaller than thereference value among a plurality of regions of the confidence map.

The processor may be further configured to: obtain a depth value of thesecond depth image as a depth value of the third depth image for a firstregion in the third depth image corresponding to a first region among aplurality of regions of the first depth image, in which a depth value ofthe first depth image is smaller than a first threshold distance, andobtain a depth value of the first depth image as a depth value of thethird depth image for a second region in the third depth imagecorresponding to a second region among a plurality of regions of thefirst depth image in which a depth value of the first depth image isgreater than a second threshold distance.

The processor may be further configured to: identify an object includedin the RGB image, identify each region of the first depth image and thesecond depth image corresponding to the identified object, and obtainthe third depth image by combining the first depth image and the seconddepth image based on a composition ratio for each of the regions.

The first image sensor may be a time of flight (ToF) sensor, and thesecond image sensor may be an RGB sensor.

According to an aspect of the disclosure, there is provided a method forcontrolling an electronic device, including: obtaining a first depthimage and a confidence map corresponding to the first depth image basedon information received from a first image sensor; obtaining an RGBimage corresponding to the first depth image based on informationreceived from a second image sensor; obtaining a second depth imagebased on the confidence map and the RGB image; and obtaining a thirddepth image based on a composition of the first depth image and thesecond depth image determined based on a pixel value of the confidencemap.

The method further includes obtaining a grayscale image for the RGBimage, and obtaining the second depth image by stereo matching theconfidence map and the grayscale image.

The method further includes obtaining the second depth image by stereomatching the confidence map and the grayscale image based on a shape ofan object included in the confidence map and the grayscale image.

The method further includes determining a first composition ratio valueof the first depth image and a second composition ratio value of thesecond depth image based on the pixel value of the confidence map, andobtaining the third depth image by combining the first depth image andthe second depth image based on the first composition ratio value andthe second composition ratio value.

The method further includes determining the first composition ratiovalue of the first depth image to be greater than the second compositionratio value of the second depth image for a first region in which apixel value is greater than a reference value among a plurality ofregions of the confidence map, and determining the first compositionratio value to be smaller than the second composition ratio value forthe region in which the pixel value is smaller than the reference valueamong a plurality of regions of the confidence map.

The method further includes obtaining a depth value of the second depthimage as a depth value of the third depth image for a first region inthe third depth image corresponding to a first region among a pluralityof regions of the first depth image, in which a depth value of the firstdepth image is smaller than a first threshold distance, and obtaining adepth value of the first depth image as a depth value of the third depthimage for a second region in the third depth image corresponding to asecond region among a plurality of regions of the first depth image inwhich a depth value of the first depth image is greater than a secondthreshold distance.

The method further includes identifying an object included in the RGBimage; identifying each region of the first depth image and the seconddepth image corresponding to the identified object, and acquiring thethird depth image by combining the first depth image and the seconddepth image based on a composition ratio for each of the identifiedregions.

Technical solutions of the disclosure are not limited to theabovementioned solutions, and solutions that are not mentioned will beclearly understood by those skilled in the art to which the disclosurepertains from the present specification and the accompanying drawings.

According to various embodiments of the disclosure, the electronicdevice may acquire distance information with improved accuracy ofdistance information for a short distance compared to the related artToF sensor.

According various embodiments of the disclosure, an autonomous vehicleor a robot may be driven smoothly based on the distance informationacquired with improved accuracy. However, the disclosure is not limitedto driving an autonomous vehicle or a robot with the distanceinformation acquired with improved accuracy. As such, according tovarious other example embodiment, the distance information acquired withimproved accuracy may be applied in other manner.

In addition, the effects obtainable or predicted by the exampleembodiments of the disclosure are to be disclosed directly or implicitlyin the detailed description of the example embodiments of thedisclosure. For example, various effects predicted according toembodiments of the disclosure will be disclosed in the detaileddescription to be described later.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for describing a method of acquiring a depth imageaccording to an example embodiment of the disclosure.

FIG. 2 is a graph illustrating a first composition ratio and a secondcomposition ratio according to a depth value of a first depth imageaccording to an example embodiment of the disclosure.

FIG. 3 is a graph illustrating the first composition ratio and thesecond composition ratio according to a pixel value of a confidence mapaccording to an example embodiment of the disclosure.

FIG. 4 is a diagram for describing a method of acquiring a third depthimage according to an example embodiment of the disclosure.

FIG. 5 is a diagram illustrating an RGB image according to an exampleembodiment of the disclosure.

FIG. 6 is a flowchart illustrating a method for controlling anelectronic device according to an example embodiment of the disclosure.

FIG. 7 is a perspective view illustrating an electronic device accordingto an example embodiment of the disclosure.

FIG. 8A is a block diagram illustrating a configuration of theelectronic device according to the example embodiment of the disclosure.

FIG. 8B is a block diagram illustrating a configuration of a processoraccording to an example embodiment of the disclosure.

DETAILED DESCRIPTION

General terms that are currently widely used were selected as terms usedin embodiments of the disclosure in consideration of functions in thedisclosure, but may be changed depending on the intention of thoseskilled in the art or a judicial precedent, the emergence of a newtechnique, and the like. In addition, in a specific case, termsarbitrarily chosen by an applicant may exist. In this case, the meaningof such terms will be mentioned in detail in a corresponding descriptionportion of the disclosure. Therefore, the terms used in embodiments ofthe disclosure should be defined on the basis of the meaning of theterms and the contents throughout the disclosure rather than simplenames of the terms.

Because the disclosure may be variously modified and have severalembodiments, specific embodiments of the disclosure will be illustratedin the drawings and be described in detail in a detailed description.However, it is to be understood that the disclosure is not limited tospecific embodiments, but includes all modifications, equivalents, andsubstitutions without departing from the scope and spirit of thedisclosure. When it is decided that a detailed description for the knownart related to the disclosure may obscure the gist of the disclosure,the detailed description will be omitted.

Terms ‘first’, ‘second’, and the like, may be used to describe variouscomponents, but the components are not to be construed as being limitedby these terms. The terms are used only to distinguish one componentfrom another component.

Singular forms are intended to include plural forms unless the contextclearly indicates otherwise. It should be understood that terms“comprise” or “include” used in the present specification, specify thepresence of features, numerals, steps, operations, components, partsmentioned in the present specification, or combinations thereof, but donot preclude the presence or addition of one or more other features,numerals, steps, operations, components, parts, or combinations thereof.

Hereinafter, embodiments of the disclosure will be described in detailwith reference to the accompanying drawings so that those skilled in theart to which the disclosure pertains may easily practice the disclosure.However, the disclosure may be implemented in various different formsand is not limited to exemplary embodiments described herein. Inaddition, in the drawings, portions unrelated to the description will beomitted to obviously describe the disclosure, and similar referencenumerals will be used to describe similar portions throughout thespecification.

FIG. 1 is a diagram for describing a method of acquiring a depth imageaccording to an example embodiment of the disclosure.

An electronic device 100 may acquire a first depth image 10 by using afirst image sensor 110. According to an example embodiment, theelectronic device 100 may acquire the first depth image 10 based on asignal output from the first image sensor 110. Here, the first depthimage 10 is an image indicating a distance from the electronic device100 to an object, and a depth value (or distance value) of each pixel ofthe first depth image may refer to a distance from the electronic device100 to the object corresponding to each pixel. According to an exampleembodiment, the depth value may be referred to as a distance value.

The electronic device 100 may acquire a confidence map 20 by using thefirst image sensor 110. According to an example embodiment, theconfidence map 20 may be referred to as a confidence image. Here, theconfidence map (or the confidence image) 20 refers to an imagerepresenting reliability of depth values for each region of the firstdepth image 10. In this case, the confidence map 20 may be an infrared(IR) image corresponding to the first depth image 10. However, thedisclosure is not limited thereto, and as such, the confidence map 20may be obtained in a different manner. In addition, the electronicdevice 100 may determine the reliability of the depth values for eachregion of the first depth image 10 based on the confidence map 20.

Meanwhile, the electronic device 100 may acquire the confidence map 20based on a signal output from the first image sensor 110. According toan example embodiment, the first image sensor 110 may include aplurality of sensors that are activated at a particular time. Forexample, the plurality of sensors may be activated at different times(i.e., at a preset time interval). In this case, the electronic device100 may acquire a plurality of image data through each of the pluralityof sensors. In addition, the electronic device 100 may acquire theconfidence map 20 from a plurality of acquired image data. For example,the electronic device 100 may acquire the confidence map 20 throughfollowing Equation 1.

[Confidence]=abs(I2−I4)−abs(I1−I3)   Equation 1

Here, I1 denotes a first image, I2 denotes a second images, I3 denotes athird image, and I4 denotes a fourth image.

Meanwhile, the first image sensor 110 may be implemented as a time offlight (ToF) sensor or a structured light sensor.

According to an example embodiment, the electronic device 100 mayacquire an RGB image 30 using a second image sensor 120. According to anexample embodiment, the electronic device 100 may acquire the RGB imagebased on a signal output from the second image sensor 120. In this case,the RGB image 30 may correspond to the first depth image 10 and theconfidence map 20, respectively. For example, the RGB image 30 may be animage for the same timing as the first depth image 10 and the confidencemap 20.

The electronic device 100 may acquire the RGB image 30 corresponding tothe first depth image 10 and the confidence map 20 by adjusting theactivation timing of the first image sensor 110 and the second imagesensor 120. In addition, the electronic device 100 may generate agrayscale image 40 based on R, G, and B values of the RGB image 30.Meanwhile, the second image sensor 120 may be implemented as imagesensors such as a complementary metal-oxide-semiconductor (CMOS) and acharge-coupled device (CCD).

The electronic device 100 may acquire a second depth image 50 based onthe confidence map 20 and the grayscale image 40. In particular, theelectronic device 100 may acquire the second depth image 50 byperforming stereo matching on the confidence map 20 and the grayscaleimage 40. Here, the stereo matching refers to a method of calculating adepth value by detecting in which an arbitrary point in one image islocated in the other image, and obtaining a shifted amount of thedetected result point. The electronic device 100 may identify acorresponding point in the confidence map 20 and the grayscale image 40.In this case, the electronic device 100 may identify a correspondingpoint by identifying a shape or an outline of the object included in theconfidence map 20 and the grayscale image 40. Then, the electronicdevice 100 may generate the second depth image 50 based on a disparitybetween the corresponding points identified in each of the confidencemap 20 and the grayscale image 40 and a length of a baseline. Accordingto an example embodiment, the length of the baseline may be the distancebetween the first image sensor 100 and the second image sensor 200.Meanwhile, when the stereo matching may be performed based on theconfidence map 20 and the RGB image 30, it may be difficult to find anexact corresponding point due to a difference in pixel values.Accordingly, the electronic device 100 may perform the stereo matchingbased on the grayscale image 40 instead of the RGB image 30.Accordingly, the electronic device 100 may more accurately identify thecorresponding point, and the accuracy of the depth information includedin the second depth image 50 may be improved. Meanwhile, the electronicdevice 100 may perform pre-processing such as correcting a difference inbrightness between the confidence map 20 and the grayscale image 40before performing the stereo matching.

Meanwhile, the ToF sensor has higher angular resolution and distanceaccuracy than the stereo sensor outside a reference distance, but mayhave lower angular resolution and distance accuracy than the stereosensor within the preset distance. Here, angular resolution refers tothe ability to distinguish two objects that are separated from eachother. According to an example embodiment, the reference distance may bea preset distance or a predetermined disclosure. For instance, thepresent distance may be a distance within 5 m from the ToF sensor. Forexample, when an intensity of reflected light is greater than athreshold value, a near-field virtual image may appear on a depth imagedue to a lens flare or ghost phenomenon. As a result, there is a problemin that the depth image acquired through the ToF sensor includesnear-field errors. Accordingly, the electronic device 100 may acquire athird depth image 60 having improved near-field accuracy compared to thefirst depth image 10 by using the second depth image 50 acquired throughthe stereo matching.

The electronic device 100 may acquire the third depth image 60 based onthe first depth image 10 and the second depth image 50. Specifically,the electronic device 100 may generate the third depth image 60 bycombining the first depth image 10 and the second depth image 50. Inthis case, the electronic device 100 may determine a first compositionratio α of the first depth image 10 and a second composition ratio β ofthe second depth image 50 based on at least one of the depth value ofthe first depth image 10 and the pixel value of the confidence map 20.Here, the first composition ratio α and the second composition ratio βmay have a value between 0 and 1, and the sum of the first compositionratio α and the second composition ratio β may be 1. For example, whenthe first composition ratio α is 0.6 (or 60%), the second compositionratio β may be 0.4 (or 40%). Hereinafter, a method of determining thefirst composition ratio α and the second composition ratio β will bedescribed in more detail.

FIG. 2 is a graph illustrating a first composition ratio and a secondcomposition ratio according to a depth value of a first depth imageaccording to an example embodiment of the disclosure. Referring to FIG.2 , the electronic device 100 may determine the first composition ratioα and the second composition ratio β based on a depth value D of thefirst depth image 10.

In the electronic device 100, for a first region R1 in which the depthvalue D is smaller than a first threshold distance Dth1 among theplurality of regions of the first depth image 10, the first compositionratio α may be determined to be 0, and the second composition ratio βmay be determined to be 1. According to an example embodiment, the firstthreshold distance may be a distance of 20 cm. That is, the electronicdevice 100 may acquire the depth value of the second depth image 50 asthe depth value of the third depth image 60 for a region in which thedepth value D is smaller than the first threshold distance Dth1 amongthe plurality of regions. Accordingly, the electronic device 100 mayacquire the third depth image 60 with improved near-field accuracycompared to the first depth image 10.

In the electronic device 100, for a second region R2 in which the depthvalue D is greater than a second threshold distance Dth2 among theplurality of regions of the first depth image 10, the first compositionratio α may be determined to be 1, and the second composition ratio βmay be determined to be 0. According to an example embodiment, the firstthreshold distance may be 3 m. That is, the electronic device 100 mayacquire the depth value of the first depth image 50 as the depth valueof the third depth image 60 for a region in which the depth value D isgreater than the second threshold distance Dth2 among the plurality ofregions.

In the electronic device 100, for a third region R3 in which the depthvalue D is greater than the first threshold distance Dth1 and smallerthan the second threshold distance Dth2 among the plurality of regionsof the first depth image 10, the first composition ratio α and thesecond composition ratio β may be determined such that, as the depthvalue D increases, the first composition ratio α increases and thesecond composition ratio β decreases. Since the first image sensor 110has higher far-field angular resolution than the second image sensor120, as the depth value D increases, the accuracy of the depth value ofthe third depth image 60 may be improved when the first compositionratio α increases.

Meanwhile, the electronic device 100 may determine the first compositionratio α and the second composition ratio β based on a pixel value P ofthe confidence map 20.

FIG. 3 is a graph illustrating the first composition ratio and thesecond composition ratio according to the pixel value of the confidencemap according to an example embodiment of the disclosure.

The electronic device 100 may identify a fourth region R4 in which thepixel value P is smaller than a first threshold value Pth1 among theplurality of regions of the confidence map 20. In addition, when eachregion of the first depth image 10 and the second depth image 50corresponding to the fourth region R4 is composed, the electronic device100 may determine the first composition ratio α as 0 and the secondcomposition ratio β as 1. That is, when it is determined that thereliability of the first depth image 10 is smaller than the firstthreshold value Pth1, the electronic device 100 may acquire the depthvalue of the second depth image 50 as the depth value of the third depthimage 60. Accordingly, the electronic device 100 may acquire the thirddepth image 60 with improved distance accuracy compared to the firstdepth image 10.

The electronic device 100 may identify a fifth region R5 in which thepixel value is greater than a second threshold value Pth2 among theplurality of regions of the confidence map 20. In addition, when eachregion of the first depth image 10 and the second depth image 50corresponding to the fifth region R5 is composed, the electronic device100 may determine the first composition ratio α as 1 and the secondcomposition ratio β as 0. That is, when it is determined that thereliability of the first depth image 10 is smaller than the secondthreshold value Pth2, the electronic device 100 may acquire the depthvalue of the first depth image 10 as the depth value of the third depthimage 60.

The electronic device 100 may identify a sixth region R6 in which thepixel value P is greater than the first threshold value Pth1 and smallerthan the second threshold value Pth2 among the plurality of regions ofthe confidence map 20. In addition, when each region of the first depthimage 10 and the second depth image 50 corresponding to the sixth regionR6 is composed, the electronic device 100 may determine the firstcomposition ratio α and the second composition ratio β so that, as thepixel value P increases, the first composition ratio α increases and thesecond composition ratio β decreases. That is, the electronic device 100may increase the first composition ratio α as the reliability of thefirst depth image 10 increases. Accordingly, the accuracy of the depthvalue of the third depth image 60 may be improved.

Meanwhile, the electronic device 100 may determine the first compositionratio α and the second composition ratio β based on the depth value D ofthe first depth image 10 and the pixel value P of the confidence map 20.In particular, the electronic device 100 may consider the pixel value Pof the confidence map 20 when determining the first composition ratio αand the second composition ratio β for the third region R3. For example,when the pixel value of the confidence map 20 corresponding to the thirdregion R3 is greater than a preset value, the electronic device 100 maydetermine the first composition ratio α and the second composition ratioβ so that the first composition ratio α is greater than the secondcomposition ratio β. On the other hand, when the pixel value of theconfidence map 20 corresponding to the third region R3 is smaller than apreset value, the electronic device 100 may determine the firstcomposition ratio α and the second composition ratio β so that the firstcomposition ratio α is smaller than the second composition ratio β. Theelectronic device 100 may increase the first composition ratio α as thepixel value of the confidence map 20 corresponding to the third regionR3 increases. That is, the electronic device 100 may increase the firstcomposition ratio α for the third region R3 as the reliability of thefirst depth image 10 increases.

The electronic device 100 may acquire the third depth image 60 based onthe first composition ratio α and the second composition ratio β thusobtained. The electronic device 100 may acquire the distance informationon the object based on the third depth image 60. Alternatively, theelectronic device 100 may generate a driving path of the electronicdevice 100 based on the third depth image 60. Meanwhile, FIGS. 2 and 3illustrate that the first composition ratio α and the second compositionratio β vary linearly, but this is only an example, and the firstcomposition ratio α and the second composition ratio β may varynon-linearly.

FIG. 4 is a diagram for describing a method of acquiring a third depthimage according to an example embodiment of the disclosure. Referring toFIG. 4 , the first depth image 10 may include a 1-1th region R1-1, a2-1th region R2-1, and a 3-1th region R3-1. The 1-1th region R1-1 maycorrespond to the first region R1 of FIG. 2 , and the 2-1th region R2-1may correspond to the second region R2 of FIG. 2 . That is, a depthvalue D11 of the 1-1th region R1-1 may be smaller than the firstthreshold distance Dth1, and a depth value D12 of the 2-1th region R2-1may be greater than the second threshold distance Dth2. Also, a 3-1thregion R3-1 may correspond to the third region R3 of FIG. 2 . That is, adepth value D13 of the 3-1th region R3-1 may be greater than the firstthreshold distance Dth1 and smaller than the second threshold distanceDth2.

When the first depth image 10 and the second depth image 50 are composedfor the 1-1th region R1-1, the electronic device 100 may determine thefirst composition ratio α as 0 and the second composition ratio β as 1.Accordingly, the electronic device 100 may acquire a depth value D21 ofthe second depth image 50 as a depth value D31 of the third depth image60.

When the first depth image 10 and the second depth image 50 are composedfor the 2-1th region R2-1, the electronic device 100 may determine thefirst composition ratio α as 1 and the second composition ratio β as 0.Accordingly, the electronic device 100 may acquire the depth value D12of the first depth image 10 as a depth value D32 of the third depthimage 60.

When the first depth image 10 and the second depth image 50 are composedfor the 3-1th region R3-1, the electronic device 100 may determine thefirst composition ratio a and the second composition ratio β based onthe confidence map 20. For example, if a depth value P3 of theconfidence map 20 is smaller than a preset value, when the first depthimage 10 and the second depth image 50 are composed for the 3-1th regionR3-1, the electronic device 100 may determine the first compositionratio α and the second composition ratio β so that the first compositionratio α is smaller than the second composition ratio β. As anotherexample, if the depth value P3 of the confidence map 20 is greater thana preset value, when the first depth image 10 and the second depth image50 are composed for the 3-1th region R3-1, the electronic device 100 maydetermine the first composition ratio α and the second composition ratioβ so that the first composition ratio α is greater than the secondcomposition ratio β. As described above, the electronic device 100 mayacquire a depth value D33 of the third depth image 60 by applying thefirst composition ratio α to the depth value D13 of the first depthimage 10, and the second composition ratio β to a depth value D23 of thesecond depth image 50.

Meanwhile, the electronic device 100 may acquire the third depth image60 by applying a predetermined composition ratio to the same objectincluded in the first depth image 10 and the second depth image 50.

FIG. 5 is a diagram illustrating an RGB image according to an exampleembodiment of the disclosure. Referring to FIG. 5 , the RGB image 30 mayinclude a first object ob1.

The electronic device 100 may analyze the RGB image 30 to identify thefirst object ob1. In this case, the electronic device 100 may identifythe first object ob1 using an object recognition algorithm.Alternatively, the electronic device 100 may identify the first objectob1 by inputting the RGB image 30 to a neural network model trained toidentify an object included in the image.

When the first depth image 10 and the second depth image 50 are composedfor the region corresponding to the first object ob1, the electronicdevice 100 may apply a predetermined composition ratio. For example, theelectronic device 100 may apply a 1-1th composition ratio α₁ and a 2-1thcomposition ratio β₁, which are fixed values, to the regioncorresponding to the first object ob1. Accordingly, the electronicdevice 100 may acquire the third depth image 60 in which the distanceerror for the first object ob1 is improved.

FIG. 6 is a flowchart illustrating a method for controlling anelectronic device according to an example embodiment of the disclosure.

The electronic device 100 may acquire the first depth image and theconfidence map corresponding to the first depth image using the firstimage sensor (S610), and acquire the RGB image corresponding to thefirst depth image using the second image sensor (S620). As a detaileddescription thereof has been described with reference to FIG. 1 , aredundant description thereof will be omitted.

The electronic device 100 may acquire the second depth image based onthe confidence map and the RGB image (S630). The electronic device 100may acquire a grayscale image for the RGB image, and acquire the seconddepth image by performing the stereo matching on the confidence map andthe grayscale image. In this case, the electronic device 100 may acquirethe second depth image by performing the stereo matching on theconfidence map and the grayscale image based on the shape of the objectincluded in the confidence map and the grayscale image.

The electronic device 100 may obtain the third depth image by combiningthe first depth image and the second depth image based on the pixelvalue of the confidence map (S640). The electronic device 100 maydetermine the composition ratio of the first depth image and the seconddepth image based on the pixel value of the confidence map, and composethe first depth image and the second depth image based on the determinedcomposition ratio to acquire the third depth image. In this case, theelectronic device 100 may determine the first composition ratio and thesecond composition ratio so that the first composition ratio of thefirst depth image is greater than the second composition ratio of thesecond depth image for the region in which the pixel value is greaterthan a preset value among the plurality of regions of the confidencemap. The electronic device 100 may determine the first composition ratioand the second composition ratio so that the first composition ratio issmaller than the second composition ratio for the region in which thepixel value is smaller than a preset value among the plurality ofregions of the confidence map.

FIG. 7 is a perspective view illustrating an electronic device accordingto an example embodiment of the disclosure.

The electronic device 100 may include the first image sensor 110 and thesecond image sensor 120. In this case, the distance between the firstimage sensor 110 and the second image sensor 120 may be defined as alength L of a baseline.

A related art stereo sensor using two cameras has a limitation in thatthe angular resolution for a long distance is lowered because the lengthof the baseline is limited. In addition, in order to increase theangular resolution for a long distance, since the length of the baselineneeds to increase, there is a problem in that the related art stereosensor is difficult to miniaturize.

On the other hand, as described above, the electronic device 100according to the disclosure uses the first image sensor 110 having ahigher angular resolution for a long distance compared to the stereosensor as described above, to acquire the far-field information even ifthe length L of the baseline does not increase. Accordingly, theelectronic device 100 may have a technical effect that it is easier tominiaturize compared to the related art stereo sensor.

FIG. 8A is a block diagram illustrating a configuration of theelectronic device according to the example embodiment of the disclosure.Referring to FIG. 8A, the electronic device 100 may include a lightemitter 105, a first image sensor 110, the second image sensor 120, amemory 130, a communication interface 140, a driver 150, and a processor160. In particular, the electronic device 100 according to the exampleembodiment of the disclosure may be implemented as a movable robot.

The light emitter 105 may emit light toward an object. In this case, thelight (hereinafter, emitted light) emitted from the light emitter 105may have a waveform in the form of a sinusoidal wave. However, this isonly an example, and the emitted light may have a waveform in the formof a square wave. Also, the light emitter 105 may include various typesof laser devices. For example, the light emitter 105 may include avertical cavity surface emitting laser (VCSEL) or a laser diode (LD).Meanwhile, the light emitter 105 may include a plurality of laserdevices. In this case, a plurality of laser devices may be arranged inan array form. Also, the light emitter 105 may emit light of variousfrequency bands. For example, the light emitter 105 may emit a laserbeam having a frequency of 100 MHz.

The first image sensor 110 is configured to acquire the depth image. Thefirst image sensor 110 may acquire reflected light reflected from theobject after being emitted from the light emitter 105. The processor 160may acquire the depth image based on the reflected light acquired by thefirst image sensor 110. For example, the processor 160 may acquire thedepth image based on a difference (i.e., flight time of light) betweenemission timing of the light emitted from the light emitter 105 andtiming at which the image sensor 110 receives the reflected light.Alternatively, the processor 160 may acquire the depth image based on adifference between a phase of the light emitted from the light emitter105 and a phase of the reflected light acquired by the image sensor 110.Meanwhile, the first image sensor 110 may be implemented as the time offlight (ToF) sensor or the structured light sensor.

The second image sensor 120 is configured to acquire an RGB image. Forexample, the second image sensor 120 may be implemented as image sensorssuch as a complementary metal-oxide-semiconductor (CMOS) and acharge-coupled device (CCD).

The memory 130 may store an operating system (OS) for controlling ageneral operation of components of the electronic device 100 andcommands or data related to components of the electronic device 100. Tothis end, the memory 130 may be implemented as a non-volatile memory(e.g., a hard disk, a solid state drive (SSD), a flash memory), avolatile memory, or the like.

The communication interface 140 includes at least one circuit and maycommunicate with various types of external devices according to varioustypes of communication methods. The communication interface 140 mayinclude at least one of a Wi-Fi communication module, a cellularcommunication module, a 3rd generation (3G) mobile communication module,a 4th generation (4G) mobile communication module, a 4th generation LongTerm Evolution (LTE) communication module, and a 5th generation (5G)mobile communication module. For example, the electronic device 100 maytransmit an image acquired using the second image sensor 120 to a userterminal through the communication interface 140.

The driver 150 is configured to move the electronic device 100. Inparticular, the driver 150 may include an actuator for driving theelectronic device 100. Also, the driver 150 may include an actuator fordriving a motion of another physical component (e.g., an arm, etc.) ofthe electronic device 100. For example, the electronic device 100 maycontrol the driver 150 to move or operate based on the depth informationobtained through the first image sensor 110 and the second image sensor120.

The processor 160 may control the overall operation of the electronicdevice 100.

Referring to FIG. 8B, the processor 160 may include a first depth imageacquisition module 161, a confidence map acquisition module 162, an RGBimage acquisition module 163, a grayscale image acquisition module 164,a second depth image acquisition module 165, and a third depth imageacquisition module 166. Meanwhile, each module of the processor 160 maybe implemented as a software module, but may also be implemented in aform in which software and hardware are combined. According to anexample embodiment, the processor 160 may execute one or moreinstructions stored in the memory to implement the various modules.However, the disclosure is not limited thereto, and as such, the modulesmay be implemented by hardware components such as circuits. According toan example embodiment, the processor 160 may include one or moreprocessors, which may include a processing unit such as a centralprocessing unit (CPU), a digital signal processor (DSP), a graphicsprocessing unit (GPU), or a machine learning processing unit.

The first depth image acquisition module 161 may acquire the first depthimage based on the signal output from the first image sensor 110.Specifically, the first image sensor 110 may include a plurality ofsensors that are activated at a preset time difference. In this case,the first depth image acquisition module 161 may calculate a time offlight of light based on a plurality of image data acquired through aplurality of sensors, and acquire the first depth image based on thecalculated time of flight of light.

The confidence map acquisition module 162 may acquire the confidence mapbased on the signal output from the first image sensor 110.Specifically, the first image sensor 110 may include a plurality ofsensors that are activated at a preset time difference. In this case,the confidence map acquisition module 162 may acquire a plurality ofimage data through each of the plurality of sensors. In addition, theconfidence map acquisition module 162 may acquire the confidence map 20using the plurality of acquired image data. For example, the confidencemap acquisition module 162 may acquire the confidence map 20 based onEquation 1 described above.

The RGB image acquisition module 163 may acquire the RGB image based onthe signal output from the second image sensor 120. In this case, theacquired RGB image may correspond to the first depth image and theconfidence map.

The grayscale image acquisition module 164 may acquire the grayscaleimage based on the RGB image acquired by the RGB image acquisitionmodule 163. Specifically, the grayscale image acquisition module 164 maygenerate the grayscale image based on the R, G, and B values of the RGBimage.

The second depth image acquisition module 165 may acquire the seconddepth image based on the confidence map acquired by the confidence mapacquisition module 162 and the grayscale image acquired by the grayscaleimage acquisition module 164. Specifically, the second depth imageacquisition module 165 may generate the second depth image by performingthe stereo matching on the confidence map and the grayscale image. Thesecond depth image acquisition module 165 may identify correspondingpoints in the confidence map and the grayscale image. In this case, thesecond depth image acquisition module 165 may identify the correspondingpoints by identifying the shape or outline of the object included in theconfidence map and the grayscale image. In addition, the second depthimage acquisition module 165 may generate the second depth image basedon the disparity between the corresponding points identified in each ofthe confidence map and the grayscale image and the length of thebaseline.

As such, the second depth image acquisition module 165 may moreaccurately identify the corresponding points by performing the stereomatching based on the grayscale image instead of the RGB image.Accordingly, it is possible to improve the accuracy of the depthinformation included in the second depth image. Meanwhile, the seconddepth image acquisition module 165 may perform preprocessing such ascorrecting a difference in brightness between the confidence map and thegrayscale image before performing the stereo matching.

The third depth image acquisition module 166 may acquire the third depthimage based on the first depth image and the second depth image. Indetail, the third depth image acquisition module 166 may generate thethird depth image by combining the first depth image and the seconddepth image. In this case, the third depth image acquisition module 166may determine the first composition ratio for the first depth image andthe second composition ratio for the second depth image based on thedepth value of the first depth image. For example, the third depth imageacquisition module 166 may determine the first composition ratio as 0and the second composition ratio as 1 for the first region in which thedepth value is smaller than the first threshold distance among theplurality of regions of the first depth image. In addition, the thirddepth image acquisition module 166 may determine the first compositionratio as 1 and the second composition ratio as 0 for the second regionin which the depth value is greater than the second threshold distanceamong the plurality of regions of the first depth image.

Meanwhile, the third depth image acquisition module 166 may determinethe composition ratio based on the pixel value of the confidence map forthe third region in which the depth value is greater than the firstthreshold distance and smaller than the second threshold distance amongthe plurality of regions of the first depth image. For example, when thepixel value of the confidence map corresponding to the third region issmaller than a preset value, the third depth image acquisition module166 may determine the first composition ratio and the second compositionratio so that the first composition ratio is smaller than the secondcomposition ratio. When the pixel value of the confidence mapcorresponding to the third region is larger than a reference value, thethird depth image acquisition module 166 may determine the firstcomposition ratio and the second composition ratio so that the firstcomposition ratio is greater than the second composition ratio.According to an example embodiment, the reference value is a presetvalue or predetermined value. That is, the third depth image acquisitionmodule 166 may determine the first composition ratio and the secondcomposition ratio so that the first composition ratio increases and thesecond composition ratio decreases as the pixel value of the confidencemap corresponding to the third region increases.

Meanwhile, the third depth image acquisition module 166 may compose thefirst depth image and the second depth image with a referencecomposition ratio for the same object. According to an exampleembodiment, the reference composition ratio may be a predeterminedration. For example, the third depth image acquisition module 166 mayanalyze the RGB image to identify the object included in the RGB image.In addition, the third depth image acquisition module 166 may apply apredetermined composition ratio to the first region of the first depthimage and the second region of the second depth image corresponding tothe identified object to compose the first depth image and the seconddepth image.

Meanwhile, the processor 160 may perform an adjustment to synchronizethe first image sensor 110 and the second image sensor 120. Accordingly,the first depth image, the confidence map, and the second depth imagemay correspond to each other. That is, the first depth image, theconfidence map, and the second depth image may be images for the sametiming.

Meanwhile, the diverse example embodiments described above may beimplemented in a computer or an apparatus similar to the computer usingsoftware, hardware, or a combination of software and hardware. In somecases, example embodiments described in the disclosure may beimplemented as a processor itself. According to a softwareimplementation, embodiments such as procedures and functions describedin the specification may be implemented as separate software modules.Each of the software modules may perform one or more functions andoperations described in the specification.

Meanwhile, computer instructions for performing processing operationsaccording to the diverse example embodiments of the disclosure describedabove may be stored in a non-transitory computer-readable medium. Thecomputer instructions stored in the non-transitory computer-readablemedium may cause a specific device to perform the processing operationsaccording to the diverse embodiments described above when they areexecuted by a processor.

The non-transitory computer-readable medium is not a medium that storesdata for a while, such as a register, a cache, a memory, or the like,but means a medium that semi-permanently stores data and is readable bythe device. Specific examples of the non-transitory computer-readablemedium may include a compact disk (CD), a digital versatile disk (DVD),a hard disk, a Blu-ray disk, a USB, a memory card, a read only memory(ROM), and the like.

Although the example embodiments of the disclosure have been illustratedand described hereinabove, the disclosure is not limited to the specificembodiments described above, but may be variously modified by thoseskilled in the art to which the disclosure pertains without departingfrom the gist of the disclosure as disclosed in the accompanying claims.These modifications should also be understood to fall within the scopeand spirit of the disclosure.

What is claimed is:
 1. An electronic device, comprising: a first imagesensor; a second image sensor; and a processor configured to: obtain afirst depth image and a confidence map corresponding to the first depthimage based on information received from the first image sensor, obtainan RGB image corresponding to the first depth image based on informationreceived from the second image sensor, obtain a second depth image basedon the confidence map and the RGB image, and obtain a third depth imagebased on a composition of the first depth image and the second depthimage determined based on a pixel value of the confidence map.
 2. Theelectronic device as claimed in claim 1, wherein the processor isfurther configured to obtain a grayscale image from the RGB image, andthe second depth image is obtained by performing stereo matching on theconfidence map and the grayscale image.
 3. The electronic device asclaimed in claim 2, wherein the processor is further configured toobtain the second depth image by performing stereo matching on theconfidence map and the grayscale image based on a shape of an objectincluded in the confidence map and the grayscale image.
 4. Theelectronic device as claimed in claim 1, wherein the processor isfurther configured to: determine a first composition ratio value of thefirst depth image and a second composition ratio value of the seconddepth image based on the pixel value of the confidence map, and obtainthe third depth image by combining the first depth image and the seconddepth image based on the first composition ratio value and the secondcomposition ratio value.
 5. The electronic device as claimed in claim 4,wherein the processor is further configured to: determine the firstcomposition ratio value of the first depth image to be greater than thesecond composition ratio value of the second depth image for a firstregion in which a pixel value is greater than a reference value among aplurality of regions of the confidence map, and determine the firstcomposition ratio value to be smaller than the second composition ratiovalue for the region in which the pixel value is smaller than thereference value among a plurality of regions of the confidence map. 6.The electronic device as claimed in claim 1, wherein the processor isfurther configured to: obtain a depth value of the second depth image asa depth value of the third depth image for a first region in the thirddepth image corresponding to a first region among a plurality of regionsof the first depth image, in which a depth value of the first depthimage is smaller than a first threshold distance , and obtain a depthvalue of the first depth image as a depth value of the third depth imagefor a second region in the third depth image corresponding to a secondregion among a plurality of regions of the first depth image in which adepth value of the first depth image is greater than a second thresholddistance.
 7. The electronic device as claimed in claim 1, wherein theprocessor is further configured to: identify an object included in theRGB image, identify each region of the first depth image and the seconddepth image corresponding to the identified object, and obtain the thirddepth image by combining the first depth image and the second depthimage based on a composition ratio for each of the regions.
 8. Theelectronic device as claimed in claim 1, wherein the first image sensoris a time of flight (ToF) sensor, and the second image sensor is an RGBsensor.
 9. A method for controlling an electronic device, comprising:obtaining a first depth image and a confidence map corresponding to thefirst depth image based on information received from a first imagesensor; obtaining an RGB image corresponding to the first depth imagebased on information received from a second image sensor; obtaining asecond depth image based on the confidence map and the RGB image; andobtaining a third depth image based on a composition of the first depthimage and the second depth image determined based on a pixel value ofthe confidence map.
 10. The method as claimed in claim 9, furthercomprises: obtaining a grayscale image for the RGB image, and obtainingthe second depth image by stereo matching the confidence map and thegrayscale image.
 11. The method as claimed in claim 10, furthercomprises: obtaining the second depth image by stereo matching theconfidence map and the grayscale image based on a shape of an objectincluded in the confidence map and the grayscale image.
 12. The methodas claimed in claim 9, further comprises: determining a firstcomposition ratio value of the first depth image and a secondcomposition ratio value of the second depth image based on the pixelvalue of the confidence map, and obtaining the third depth image bycombining the first depth image and the second depth image based on thefirst composition ratio value and the second composition ratio value.13. The method as claimed in claim 12, further comprising: determiningthe first composition ratio value of the first depth image to be greaterthan the second composition ratio value of the second depth image for afirst region in which a pixel value is greater than a reference valueamong a plurality of regions of the confidence map, and determining thefirst composition ratio value to be smaller than the second compositionratio value for the region in which the pixel value is smaller than thereference value among a plurality of regions of the confidence map. 14.The method as claimed in claim 9, further comprises: obtaining a depthvalue of the second depth image as a depth value of the third depthimage for a first region in the third depth image corresponding to afirst region among a plurality of regions of the first depth image, inwhich a depth value of the first depth image is smaller than a firstthreshold distance, and obtaining a depth value of the first depth imageas a depth value of the third depth image for a second region in thethird depth image corresponding to a second region among a plurality ofregions of the first depth image in which a depth value of the firstdepth image is greater than a second threshold distance.
 15. The methodas claimed in claim 9, further comprises: identifying an object includedin the RGB image; identifying each region of the first depth image andthe second depth image corresponding to the identified object, andacquiring the third depth image by combining the first depth image andthe second depth image based on a composition ratio for each of theidentified regions.